Beware the impact factor

The research community is under increasing pressure to document the impact of its activities. One of the root causes of this pressure is the entirely reasonable desire on the part of the funders of research—and ultimately society at large—to know what benefits they are accruing on their investments. Done judiciously, an ongoing assessment of the impact of research would be a good thing for all involved: society, funding agencies, and the research community itself. Done injudiciously, such assessments can be powerfully counterproductive. Judging impact begs the obvious but often ignored question: impact on whom or on what? Assessments require metrics on which to draw conclusions about impact. In the research community, the h factor (Hirsch 2005) is usually used to assess the impact of an individual’s research efforts and has been extended to institutions (Prathap 2006) and journals (Moussa and Touzani 2010), while the impact factor has become another metric for judging the quality of journals in which research results are published (Garfield 2006). One recent example of the pressure to use these metrics comes from the governing board of a Norwegian development and aid research program. The Norway-Global Partner (NORGLOBAL) program is sponsored by the Norwegian Research Council and the Norwegian Agency for Development Cooperation (Norad). It is very broadly interdisciplinary in nature, and as such presents special challenges in terms of impact assessment. At a recent meeting of the governing board of the NORGLOBAL program (Oslo, 26 January 2016), representatives from Norad expressed the desire to include the impact factor of the journals in which results from the research funded through the program are published as part of final reports for projects. On the surface, this would seem to be a reasonable request. On deeper consideration, this may be leading us down the wrong path. Like the Gross Domestic Product (GDP) for measuring economic impact, these metrics have become defaults in terms of assessing the quality and impact of research efforts. In the 1934 report that defined the concept of GDP (Kuznets 1934), Simon Kuznets and co-authors wrote ‘‘A student of social affairs who is interested in the total productivity of the nation, including those efforts which, like housewives services, do not appear on the market, can therefore use our measures only with some qualifications.’’ Similarly, I suggest that h and impact factors should only be used to assess research impact ‘‘with some qualifications’’. This criticism is not new; the utility of the h factor has been widely discussed in the literature (Lehmann et al. 2006; Bornmann and Daniel 2008). Criticism of these indices includes the difficulty of comparisons across disciplines, comparing academics at different stages in their careers, and the placement of individuals in the author list. Using the h or impact factors is particularly problematic in areas such as development research, since they are dominated by journals (and the associated logic, cognitive values and perspectives) from the global North published in English. I can provide one example of the limitations of these metrics for determining impact from personal experience. It was a competition for my time between two activities; one activity was helping a PhD student write a series of papers on a new analytical technique for determining the sources of biogenic aerosol particles. The work involved very detailed analytical chemistry, and was focused on a specific issue that has been confounding atmospheric chemists. One of the papers (Gonzalez et al. 2014) was selected as a ‘‘hot’’ article by the journal Environmental Science: Processes and Impacts (which, since we are on the topic, has an impact factor of 2.171). This paper represented sterling scientific work on the part of the then PhD